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Abstract 

Background: Preparing medical students for the takeover or the start-up of a medical practice is an important 
challenge in Germany today. Therefore, this paper presents a computer-aided serious game (eMedOffice) developed 
and currently in use at the RWTH Aachen University Medical School. The game is part of the attempt to teach 
medical students the organizational and conceptual basics of the medical practice of a general practitioner in a 
problem-based learning environment. This paper introduces methods and concepts used to develop the serious 
game and describes the results of an evaluation of the game's application in curricular courses at the Medical 
School. 

Results: Results of the conducted evaluation gave evidence of a positive learning effect of the serious game. 
Educational supervisors observed strong collaboration among the players inspired by the competitive gaming 
aspects. In addition, an increase in willingness to learn and the exploration of new self-invented ideas were 
observed and valuable proposals for further prospective enhancements were elicited. A statistical analysis of the 
results of an evaluation provided a clear indication of the positive learning effect of the game. A usability 
questionnaire survey revealed a very good overall score of 4.07 (5=best, 1= worst). 

Conclusions: We consider web-based, collaborative serious games to be a promising means of improving medical 
education. The insights gained by the implementation of eMedOffice will promote the future development of more 
effective serious games for integration into curricular courses of the RWTH Aachen University Medical School. 

Keywords: Computer-Assisted Instruction/methods, Games, Experimental, Teaching/methods, Education, Medical, 
Undergraduate/methods, User-Computer Interface 



Background 

The German healthcare system faces a downturn in the 
number of general practitioners starting or taking over a 
practice, especially in the rural areas [1,2]. Financial and 
administrative incentives have been introduced to im- 
prove the situation. In Germany most general practi- 
tioners own their medical practice or at least most of 
their equipment. They act as entrepreneurs and take the 
full economic risk. Recent reforms in the German health- 
care system have granted patients more freedom to 
choose a general practitioner and, as a result, encourage 
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competition [3,4]. Nonetheless, medical education is chal- 
lenged to play its part by preparing medical students to 
cope with the increasingly risky and cost-intensive ven- 
ture of a practice takeover or start-up. In the context of 
economics studies business games have been proven to 
support training in the skills necessary for starting up an 
enterprise [5,6]. Enhanced by computer-based communi- 
cation and simulations, similar game-based learning inter- 
ventions have evolved to a new category of serious games. 
Recently, serious games have been successfully applied to 
some fields of medical education [7,8]. The challenge of 
adequately preparing medical students for future practice 
start-up provides an excellent opportunity for further 
developing and testing collaborative aspects of serious 
games in medical education. 
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A general-purpose business game (or business simula- 
tion) has been defined as a complex man-made environ- 
ment, where participants are able to gain experiences 
when acting in a simulated reality [9,10]. Yet there is 
open discussion concerning the definition of the term 
serious game. A relatively broad definition of a serious 
game is any computerized game whose chief mission is 
not entertainment and all entertainment games which 
can be reapplied to a different mission other than enter- 
tainment' [11]. In contrast to teacher-centred learning 
approaches, which primarily rely on the control and ac- 
tivity of the teacher, serious games are considered as in- 
herently learner-centred and focusing on the needs of 
learners [12]. Experts in the field of serious games pre- 
dict a continuous growth in this research area [13,14]. 
Recently, research activity in the field of serious games 
has indeed markedly increased and has especially 
addressed their application in medical environments 
[15]. Ritterfeld and colleagues (2009) found that 8% of 
all existing serious games deal with a health- or 
medicine-related topic [16]. Medical learners showed a 
high acceptance of serious games as shown in several 
studies: The INMEDEA simulator, which presents virtual 
case studies and adopts an open, nonlinear problem- 
oriented problem-solving approach situated in a virtual 
hospital, was judged as a good or very good teaching 
method by 76% of 70 test candidates [17]. As a second 
example the Junior Doctor Simulator (JDoc), an inter- 
active, 3D, third-person view prototype focusing on the 
training of doctor-patient interactions, was considered 
to provide valuable experiences of patient interactions 
and routines of a hospital ward to junior doctors [18]. 
Evaluation results suggest a significant impact of serious 
games' stimulation of conceptual thinking on the learn- 
ing outcome [19]. JDOC and INMEDEA teach common 
procedures of the everyday work of doctors in different 
contexts and focus on doctor-patient interaction and 
medical diagnosis. Recent projects have tried to close 
the gap between computer-supported learning and real- 
ity by introducing learning support to mobile devices 
[20,21]. Improved collaboration induced by serious 
games in the nursing context was stressed some time 
ago [22]; in a more recent review of virtual 3-D medical 



learning environments the collaborative aspect was iden- 
tified as a major strength associated with several game- 
oriented medical learning approaches based on virtual 
worlds [7] . Following the idea of applying a serious game 
approach to prepare medical students for future practice 
start-up the 'Business Game: How to start a Medical 
Practice' was set up as a joint project of the Division of 
General Medicine and the Department for Knowledge- 
based Systems of the Institute for Medical Informatics of 
the RWTH Aachen University Medical School. The 
learning scenario consists of different learning and 
computer-aided game phases covering an online search, 
rough and fine-grained planning of funding and the 
detailed planning of required medical equipment. A spe- 
cial computer-assisted module of the business game is 
the interactive, collaborative serious game eMedOffice. 
The game eMedOffice focuses on practical exercises 
guiding the participants towards strategies for an 
optimization of the interior design, workflows, equip- 
ment, components and furnishing arrangements of a 
medical practice. The goal of eMedOffice is to provide 
fun access to the necessary theoretical knowledge to 
promote relevant practical problem-solving skills of the 
learners. In this work we present the theoretical ap- 
proach, technical realization, and evaluation results of 
eMedOffice that evidence the positive effect of this 
learning game. 

Methods 

Learning Design 

Research on business games yielded an abstract game 
model, which has been applied to the design of eMedOf- 
fice. The model includes the input, process and outcome 
objectives (Figure 1). As an input objective, the model 
integrates instructional content and specific game char- 
acteristics into eMedOffice. The input triggers a game 
cycle that consists of the users judgements and behav- 
iour, and the systems feedback. A variable number of 
iterations of this cycle during the process are possible. 
These iterations are similar to the cycle of expertise 
which describes the approach of experts to creating new 
strategies to solve problems [23]. The application of the 
cycle of expertise fosters a motivating gaming experience 
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Figure 1 Input-Process-Outcome Game Model is an inherent model of instructional games adapted from Garris and colleagues (2002) [10]. 
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and encourages the exploration of self-invented ideas. 
Finally, the game cycle ends with a debriefing that evalu- 
ates the learning process and ensures a positive learning 
outcome. As seen from the perspective of the learner, 
the learning scenario contains a sequence of different 
learning phases (Figure 2): it starts and ends on a meta 
level introducing the game and evaluating the process. 
On the game level, the learners are involved in the prep- 
aration of the game and their individual gaming activ- 
ities (e. g. understanding the games' rules or choosing 
room functionalities). The described cycle of expertise 
resides in the active game phase. As game characteristics 
we adopted aspects of successful computer entertain- 
ment games like challenge, control, fantasy, clear rules 
and clear objectives in order to increase the willingness 
to learn. Some principles of good learning games serve 
as learning concepts of eMedOffice: Empowered learners 
(player is co-designer, active agent, with own identity in 
a customizable game play), problem-solving (problems 
are well-ordered and information is given when needed), 
and understanding (skills and strategies to be learned 
are embedded into a larger context and acquired by 
learning through experience) [24]. Each of these princi- 
ples consists of more sub-principles. One example of a 
sub-principle for empowered learners is that learners 
feel like active agents and not just like passive consu- 
mers; another example is the cycle of expertise which is 
part of the problem-solving principle. 

Knowledge acquisition, representation and maintenance 

The game has been developed in close cooperation with 
medical students and the Division of General Medicine of 
RWTH Aachen University Medical School. In preliminary 
work, the implemented step-wise structure and learning 
objectives of the 'Business Game: How to start a Medical 
Practice' (eMedOffice) were acquired. Oriented on a busi- 
ness game used in economic sciences at RWTH Aachen 
University named EasyStartup, the requirements were 
analysed in several meetings with students and experts in 
medical practice design. Using a top-down approach, the 
game steps were iteratively refined to provide a conveni- 
ent computer-supported way to achieve the learning 
objectives in an explorative manner. Acquisition of 
required special knowledge on how to characterize a 



medical practice for eMedOffice with sound workflows 
and efficient design was part of an on-going dissertation 
project (Ruhr 2012, to be published). In particular, it was 
the main task of the dissertation to find rules usable for 
scoring and to verify the external validity of the scoring 
system. We used a pattern language that we tailored to 
our particular needs to gather and transfer the specific 
knowledge from the medical domain to an operational 
format. This acquired domain knowledge was represented 
and formalized in an ontology. A short and general defin- 
ition of an ontology is given by Gruber and colleagues: 
An ontology is an explicit specification of a 
conceptualization [25]. The ontology editor Protege was 
used for acquisition and maintenance of the semantic 
model [26]. We developed a transformation for the repre- 
sentation of non-branched workflows to concepts of an 
ontology that allowed a visual simulation of many com- 
mon non-branched workflows of general practitioners. By 
using the A* path-finding algorithm we integrated a simu- 
lation of agents on the ground plan [27]. 

Scoring algorithm 

Conditions for a medical practice with sound workflows 
and adequate furnishing of a general practice are forma- 
lized as ontology restrictions. An evaluation algorithm 
verifies these restrictions by calculating an appropriate 
score. Examples of integrated rules formalized as restric- 
tions are All beds must be usable' and In the briefing 
room there is a medicine cabinet'. Many rules check the 
existence of furniture and equipment items in a specific 
room. Others check useful composition and usability of 
placed furniture and equipment items. A score is 
assigned to each of these rules. If a player's solution 
meets the rule, the assigned score is added to the 
player's total score. The score calculation is done by a 
conformity check of the restrictions with a student's so- 
lution which is translated to an instance of the ontology- 
based object model. The result of this conformity check 
is a list of rules that the current student's solution com- 
plies with and those with which it does not. A scoring 
algorithm calculates the final score based on the former 
rules according to a score table. For this calculation, we 
used a custom language developed for the purpose that 
is able to address the specific conditions of an ontology. 
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Figure 2 Game phases of eMedOffice adapted from Ulrich (2006) [29]. 
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Learning objectives 

The learners should realize the fundamental relationship 
between the needs of patients and staff arising during 
typical processes in a medical practice on the one hand 
and the requirements concerning (a) the assignments of 
functions to rooms and (b) the arrangement of furniture 
and technical equipment on the other hand. The partici- 
pants should gain orientation on the usage of rooms, 
furniture and equipment during the daily routine. 
Players should develop useful furniture and equipment 
compositions in the different rooms. In this game 
players do not learn factual knowledge by heart. Instead 
players are encouraged to explore self-invented solutions 
to problems when arranging a medical practice. The 
game is intended to teach observation and analytical 
skills as well as planning and deductive skills in the con- 
text of furnishing a medical practice by using immediate 
feedback of simulated patients. Thus, as a primary goal 
the participants should learn to react to patients' (and 
staffs) problems caused by suboptimal arrangement of 
rooms, furniture items and equipment items. This goal 
reflects the fact that requirements may change dynamic- 
ally over time. The participants should adopt a patient- 
and process-oriented approach to adjusting the 
organization of a medical practice. 

Hardware / software platform 

Evaluations of the game used a virtual linux server ma- 
chine with one gigabyte of RAM and 50 megabytes of 
free hard disk. For execution on the server side the game 
depends on common web techniques like a web server 
and a standard database that supports SQL. Clients of 
the game use a standard web browser without any 
requirements for extensions. To play eMedOffice, an ac- 
tive network connection between client and server is 
required. For a smooth visualization of integrated work- 
flows the client should have at least a 1.0 GHz CPU. 
Thus, the hardware requirements do not exceed the 
standards of available computer hardware. 

Curricular context 

As part of a revised six-year medical curriculum (Aa- 
chen Model Curriculum Medicine'), the game is embed- 
ded into the compulsory elective part (the so-called 
Individual Qualification Profile - IQP). The game is part 
of an IQP module addressing family medicine. Students 
may choose the module between the third and the sixth 
year. A group of 15 to 25 students regularly takes part in 
the module each semester. 

Evaluation design 

We performed a quantitative summative evaluation con- 
sisting of two interventions. The evaluation was con- 
ducted in the compulsory elective ICP module 'family 



medicine' at the RWTH Aachen University Medical 
School during the four semesters between autumn 2009 
and summer 2011. Thus, participants who took part in 
the IQP automatically participated in the evaluation. 
The number of participants varied from seven to 12 of 
mixed gender each semester. All participants attended 
both interventions. Each evaluation session took exactly 
90 minutes. As computer techniques and online learning 
programs are integrated into many courses of the Med- 
ical School in Aachen, students are familiar with web- 
based techniques. Therefore, the participants were 
expected to deal with common computer handling and 
working with a web browser. The students sat next to 
each other and every participant had his/her own com- 
puter to play eMedOffice. Because of this constellation, 
all participants had the chance to look at the interim 
results of their immediate neighbours. A usability ques- 
tionnaire for systematic determination of the quality of 
the serious game was the first intervention (see Appen- 
dix A, Appendix B). It was our intention to discover the 
answer to the central question 'Does the quality of the 
serious game meet the requirements of a practical real- 
world learning application?'. To answer this question all 
participants of the first intervention had to fill in an 
evaluation sheet containing 22 scale items from five 
(best) to one (worst) after playing the serious game. We 
chose questions from an evaluation of learning software 
published by Holzinger in 2003 [28]. In order to test the 
consistency of results of this first intervention we com- 
puted an intra-class correlation coefficient (ICC) using 
absolute agreement of two-way random measures. By 
using 22 scale items in the usability questionnaire we 
were able to target specific assets of the serious game. 
One of the questions put to the target audience was Is 
the software appropriate for the target audience?'. An- 
other question addressed the loading time with Is the 
loading time satisfactory?'. For an overview of all ques- 
tions see Appendix A. Not all participants filled in the 
page of this first intervention because two classes used 
another page that was not compatible. As a second 
intervention, we conducted a self-report quantitative 
evaluation to find out if the serious game was able to 
support learning processes (see Appendix B). This inter- 
ventions central question was: 'Does the serious game 
help medical students to become familiar with the inter- 
ior furnishings of a medical practice?'. Each participant 
had to fill in two self-report evaluation sheets covering 
the same seven items. The self-report used scale items 
from 5 (best) to 1 (worst). Table 1 lists the scale items 
for self-report. The first sheet had to be completed be- 
fore the serious game could begin. Thus the first sheet 
represented a self-report (pre) of the knowledge of the 
participants without knowing the concrete learning con- 
tent of the serious game. After the training, participants 
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Table 1 Self-report items 

St. no. Statement 



1) I am able to name important rooms 

2) I am able to name important furniture 

3) I am able to name important equipment 

4) I am able to estimate the financial outlay for interior furnishing 

5) I know the target rooms for placing furniture correctly 

6) I know how the furniture in specific rooms should be equipped 

7) I have an idea about good arrangements of furniture 



The first column shows the statement number used as a reference to the different 
scale items. The second column statement names the used sentence for self-report. 

had to fill in the second sheet that covered two self- 
reports. One of those was a retrospective self-report 
(retro), so the participants had to self-assess their prior 
knowledge before the training again. The second self- 
report on the second sheet was an assessment (post) 
addressing their knowledge after playing the serious 
game. All three reports are integrated into the action se- 
quence schedule of the learning session (Table 2). Each 
intervention was supervised by an educational adviser 
who had expert experience of the technical aspects of the 
serious game and basic knowledge on the medical infor- 
mation formalized in the games knowledge base. Besides 
the evaluation sheet, the oral feedback of the participants 
was recorded by the educational supervisor to provide an 
additional source of qualitative information for evaluation. 
For analysis of the second intervention the statistical 
method repeated measures analysis of variance (ANOVA) 
paired t-test (alpha = 0.05) was used. The null hypothesis 



states that no differences between the test groups exist 
and all groups are equal. 

Implementation 

The game eMedOffice is an interactive, web-based, 2D, 
bird's eye view simulation of a medical practice proto- 
type. Its purpose is to teach optimization of interior de- 
sign, including the furnishing as well as equipment 
components of a medical practice of a general practi- 
tioner. It is playable in every standard web browser and 
hence available anytime at any location where an inter- 
net connection is accessible. Constructed as a rich inter- 
net application that uses a live-connect technique to 
communicate with the web server, it offers continuous 
game play that is not interrupted by interfering web page 
loads while one is using standard web techniques. First, 
all participants have to register with a free-to-choose 
public players name. After registration the active game 
starts. Three successive game phases characterize the ac- 
tive game. During the Introduction Phase the learning 
objectives, principles and procedures of the serious game 
are presented. The introduction utilizes many in-game 
pictures to describe possible situations as an example. 
Additionally, this phase is supported by an educational 
adviser who explains the learning objectives orally and 
answers general questions. The game is played in the fol- 
lowing Execution Phase. In this phase players start to as- 
sign appropriate room functionalities, placing interior 
furniture and equipment onto the ground plan. A player 
can virtually open his/her medical practice at any time. 
Opening his/her medical practice starts a simulation. 



Table 2 Timing chart table 

Phase Action Duration (min.) 

Introduction Students arrive at the training room and sit down 5 

Educational supervisor greets students and gives an overview of the agenda 3 

Students fill in the first evaluation sheet self-report (pre) 3 

Educational supervisor collects first evaluation sheets 2 

Execution Educational supervisor gives brief instructions on how to start eMedOffice 3 

Students start web browser, navigate to the correct web address, register and log into eMedOffice 3 

Students read the introduction 5-10 

Students play eMedOffice 45 

Educational supervisor tells students to stop game play 1 

Debriefing Students stop playing and discuss eMedOffice results 5-10 

Evaluation Students fill in second evaluation sheet: self-report (retro and post) 3 

Students fill in usability questionnaire 5 
Educational supervisor collects second evaluation sheet as well as usability questionnaire and says goodbye to students 2 

Students leave the training room 1 



The first column phase shows the phase of eMedOffice in which the action takes place. The column action details the planned action. The last column duration 
presents the estimated time in minutes for performing the planned action. 
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During the simulation virtual agents of the doctor, doc- 
tor's assistants, patients and other visitors follow com- 
mon workflows and use the placed furniture and 
equipment. If an agent cannot fulfil his/her operation 
because of a missing requirement a speech bubble 
appears that describes the requirement in the form of 
first-person statements. In this situation a player can 
stop the simulation to plan and integrate a solution to 
the problem. Players are free to choose the number of 
repetitions of this enhancement procedure. In the final 
Evaluation Phase a detailed evaluation of the current 
session is presented. It includes a detailed report of the 
used furniture and equipment with overall costs and 
the price of every single item used. Extensive point 
ranking gives information on which rule accounts for 
how many points. If a rule did not comply a hint pro- 
vides possible improvements for the next time. At any 
time of game play a public player ranking is visible to 
all players. This ranking shows the relative distance be- 
tween the players according to their current score. 
Additionally, the total money spent on interior furnish- 
ing and equipment components is presented. The rank- 
ing can be projected to a screen to drive competition 
between the students. 

Results 

This section presents results concerning the game de- 
sign. In-game screenshots show the user interface as well 
as the game design. In the last part of this section we de- 
scribe the evaluation results. 

Game play and design 

The game eMedOffice consists of a web-based learning 
interface and a separate administration user interface. 
The learner s user interface (UI) provides access to the 



game itself. Accordingly, the administration UI supplies 
tools to manipulate the serious games configuration 
parameters and knowledge base settings and provides a 
real-time monitor of a running game. Thus, an adminis- 
trator is aware of players facing difficulties and hence is 
able to provide individual support by giving specific ver- 
bal hints. After registration, the game starts with an 
introductory screen (Figure 3). The layout of the website 
is set up to please the eye without compromising usabil- 
ity. At the top left side of the screen is the logo of eMe- 
dOffice. At the top middle is a step-wise history bar that 
indicates which phases are already finished and which 
will be next. The history bar is intended to provide a 
general orientation for the user. On the top right side is 
a login/logout area with the current user s name. In case 
of a log-out, a player s last position in the game is saved 
and can be retrieved when the player logs in again. In 
the introduction players have the choice of navigating 
through the content step-by-step by topic or starting the 
game without consulting introductory descriptions. 
Thus, the structure of the introduction supports the 
various reading and learning habits of players. The first 
page of the introduction describes the learning objec- 
tives. The game starts as soon as the players finish the 
introduction and switch to the playing phase (Figure 4). 
In this phase the players assign general functionalities to 
the rooms, select and place interior furnishings and sup- 
plement furniture items with equipment like blood pres- 
sure meters or stethoscopes. All these operations can be 
done with the action bar located on the left side of the 
working space. The ground plan of the medical practice 
is located on the right side of the working space. In this 
example the players have already assigned room func- 
tionalities and placed some furniture items on the 
ground plan. At any time during game play players may 
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Welcome to eMedOffice of "Business Game: Medical practice foundation" 



Learning objective of this serious game is to provide a playfully access to basic knowledge in designing a medical practice. 
Mdu will learn: 



• interior furnishing and equipment components, 

• room functionalities, 

• appropriate arranging interior furnishing, 

• analyse visitor behaviour and derive improvements. 

• learn financial effort for interior components. 



Visit Introduction Start serious game 



Proof of concept. "Business Game: Medical practice foundation" 

Figure 3 Learning objectives are presented in the introduction screen. 
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Figure 4 Screenshot: Common game play in an advanced stage of a running game. 



decide to open their medical practice. An opening of the 
medical practice starts a simulation of agents that repre- 
sent patients, medical doctors and medical doctors' 
assistants who use the interior furnishings and interact 
with each other. When opening a medical practice, 
players are informed by the game that they can open 
and close the medical practice as often as they want. A 
running simulation of randomly selected and concurrent 
common workflows of a medical practice is depicted in 
Figure 5. Patients are represented by moving red dots, 
medical doctors assistants are shown by a face wearing 
a nurse's hat and doctors are represented by a face with 



white hair. Whenever an agent detects a problem like a 
missing furniture item or equipment component while 
executing a workflow-task a speech bubble appears. In 
this speech bubble the agent uses a first-person statement 
to report that something is missing or cannot be done. In 
the case depicted in Figure 6 the doctor s agent is unable 
to proceed with an eye test because the eye test board is 
missing. The players are invited to stop the running simu- 
lation and add an eye test board to the examination room. 
Subsequent to this improvement the players can restart 
the simulation. When players decide to end game play an 
evaluation of their result is presented (Figure 6). This 
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2,000.00 € 


2,000.00 € 


1 Paiting book 


20.00 € 


20.00 € 


Rule Points 




1 Laborary 


1,000.00 € 


1,000.00 € 


1 Ophthalmoscope 


250.00 € 


250.00 € 


„ Alle Betten im Funktionsraum sind verwendbar 


0/5 




1 Medicine cupboard 


1, 800.00 € 


1,800.00 € 


1 Otoscope 


250.00 € 


250.00 € 


y All restrooms do have a toilett and a washbasin 


5/5 




6 Medium flower pot 


50.00 € 


300.00 € 


1 Reflex hammer 


20.00 € 


20.00 € 


y Number of chairs at reception is all right 


3/3 




2 Trash can 


15 00€ 


30.00 € 


3 Collection of magazines 


50.00 € 


1 50.00 € 


y Number of chairs in examination room is all right 


3/3 




3 Examination table 


300.00 € 


900.00 € 


1 Piggy bank 


10 00 € 


10.00€ 


y Number of chairs in waiting room is all right 


3/3 




8 Rack 


1 50.00 € 


1, 200.00 € 


1 Toys 


150.00 € 


1 50.00 € 


y It exists a fire distinguisher 


5/5 




1 Sonography 


1 0,000.00 €1 0,000.00 € 


1 Stethoscope 


1 80.00 € 


1 80.00 € 


y Reception is visible from the main entry door 


5/5 




1 Aeroplethysmograph 


2,000.00 € 


2,000.00 € 


1 Tuning fork 


1 00.00 € 


1 00.00 € 


y Filing cabinet is at reception 


m 




20 Chair 


1 20.00 € 


2,400.00 € 


5 Phone 


1 00.00 € 


500.00 € 




3/3 




2WC 


300.00 € 


600.00 € 




1 1,860.00 € 


v'Atable in examination room 






2Water dispenser 


380.00 € 


760.00 € 








^Atable in consulting room 


5/5 




2 Changing cubicle 


50.00 € 


1 00.00 € 








0 A table in waiting room 


0/3 




3 First aid cupboard 


350.00 € 


1,050.00 € 








y A examination table in consulting room 


3/3 




1 Scales 


1, 000.00 € 


1,000.00 € 








y It exists a storage room 


3/3 




3Washbasin 


1 00.00 € 


300.00 € 








0 ECG and ergometry are in examination room 


0/5 






36.035.00 € 








b ; 


42/57 




Total price: 47.895.00 € 










c 



Figure 6 Screenshots: Evaluation phase. 

^ ) 



evaluation consists of a short summary of the games 
results (Figure 6 a). A detailed view of the rules describes 
the used scoring system and the results (Figure 6 b). 
When the players did not comply with a rule a small bub- 
ble appears right behind the score. This bubble provides 
access to a pop-up text providing a hint on how to do bet- 
ter next time. A detailed list of used interior furniture 
items and equipment component items with single and 
total prices is provided in the financial overview (Figure 6 
c). During game play a public player ranking is projected 
on a screen visible to all players at all times (Figure 7). 
The ranking provides information about current rank, fi- 
nancial outlay and distance from the other players accord- 
ing to points achieved. The financial outlay sums all 
money spent on furniture and equipment items and is 
presented to the participants, but is not used for ranking 



or scoring calculations. The public ranking is refreshed 
every 30 seconds. 

Evaluation results 
Measurements 

In total 41 students (N = 41) participated in the qualifica- 
tion profile and used the proposed game eMedOffice. 
Analysis of usability aspects of the first intervention (N = 
27) revealed high overall usability of the proposed game 
(4.07 points on average on a scale of one to five) (see Fig- 
ure 8 for detailed results). For an average reliability (ICC) 
value we computed 0.91 which indicates that the recorded 
survey data can be generalized. The second intervention 
of the evaluation (N = 41) showed indications of the posi- 
tive effect of the proposed game. A statistical analysis 
showed a significant difference between the pre and post 



Current Ranking 



Pos. 


Playei name 


Financial effort Rel. distance 


1. 


Arztin 


48000 CZ 


2. 


Nil, ik 


44000 


3. 


Jule 


48000 ( 


4. 


Maike 


55000 


5. 


Lulsa 


45000 


6. 


superdoc 


47000 


7. 


MadMax 


55000 


8. 


Catherine 


44000 1 


9. 


Irina 


47000 1 


10. 


JanM 


59000 rr 


11. 


Miiller 


50000 



Refresh in 21 seconds 

Figure 7 Screenshot: Example of a player ranking. 
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Target audience 
Loading time 
Screen layout 
Screen design 
User guidance 
Text readability 
Symbol identification 
Motivation 
Learning atmosphere 
Curiosity 
Positive surprises 
Humor 
Creativity 
Difficulty 
Feedback 
Precise learning goal 
Reachability of learning goal 
Practise 
Varieties 

Different knowledge transfer 
Incidental learning 
Problem solving competence 



Figure 8 Mean results of usability questionnaire. 



Results (average) 



as well as retro and post groups. This disproves the null 
hypothesis, revealing a significant difference between pre 
and post groups as well as retro and post groups. An over- 
view of the computed significance levels between pre, 
retro and post groups is given in Table 3. It is evident that 
the mean values of the post groups are higher than those 
of the pre groups. Based on the results presented in 
Table 3 the null hypothesis was rejected. Hence we con- 
clude that the players consider the game as a way to im- 
prove their abilities and knowledge. This result is an 
immediate indication of the positive effect of the serious 



Table 3 Comparison of significances and means of groups 





pre 






retro 






post 






St 


Sig r 


sig P 


171 e 


Sig e 


sig P 


171 r 


Sig e 


Sig r 


m p 


1) 


.512 


< .001 


3.794 


.512 


< .001 


3.714 


< .001 


< .001 


4.512 


2) 


.225 


< .001 


3.512 


.225 


< .001 


3.686 


< .001 


< .001 


4.410 


3) 


1.00 


< .001 


3.410 


1.00 


< .001 


3.456 


< .001 


< .001 


4.315 


4) 


.030 


< .001 


2.282 


.030 


< .001 


2.714 


< .001 


< .001 


3.769 


5) 


.571 


< .001 


3.076 


.571 


< .001 


3.294 


< .001 


< .001 


4.156 


6) 


.023 


< .001 


2.923 


.023 


< .001 


3.342 


< .001 


< .001 


4.290 


7) 


.160 


< .001 


2.948 


.160 


< .001 


3.323 


< .001 


< .001 


4.210 



The first column statement (Sr.) shows the index number of the used 
statements that were introduced in chapter evaluation design. Columns 
Sig {e , r , P } show computed p-values of the paired t-test for pre (Sig e ), retro 
(Sig r ) and post (Sig p ) groups. Columns m {e< r< p} show the computed mean 
values of self-reports according to statements 1-7 (see Table 1). 



game. Two participants stated that they started the learn- 
ing course with expert knowledge and finished the train- 
ing with no knowledge. Moreover, they chose the worst 
rating without giving any free textual feedback. Thus, we 
rated the evaluation of those two participants as a general 
protest and excluded them from our statistical analysis. 
The protest could have been provoked by the enormous 
number of course evaluations in which students are asked 
to take part during a semester. 

Observations 

The observations of the educational supervisors 
revealed that the public ranking strongly encouraged 
players to communicate with each other and fostered 
competitive cooperation. One opening question asked 
very often in player conversations was: 'What did you 
do to improve your score?'. Players started to revise 
their individual solutions critically and initiated active 
and valuable discussions. Some players compared their 
individual solutions in detail to find out what made one 
solution better than another. At the same time players 
developed strong ambitions to enhance their current 
solution and some stayed beyond the scheduled time of 
the course or asked for later access from home to im- 
plement further improvements. Thus, in the light of 
observations by educational supervisors during evalua- 
tions of the game it seems to enhance problem-solving 
competence of the participants in a motivated and 
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collaborative manner. The supervisors observed that 
the participants repeated the feedback cycle of furnish- 
ing the practice, observing the simulation, stopping the 
simulation and improving the arrangements about three 
to six times during the course. After debriefing and 
evaluation, some students asked for more time to en- 
hance their solution. For that reason we made the game 
available on the internet for attendees of the course, 
some of whom did continue to use it. Students playing 
eMedOffice are depicted in Figure 9. Visible to all 
players, the public ranking is projected with a beamer 
to a screen on a wall. The public ranking is present in 
the background of Figure 9 a. The seating arrangement 
of students is illustrated in Figure 9 b. Students sit in 
rows and can look at the screen of their immediate 
neighbours. Those students depicted in Figure 9 b have 
to turn around to see the public ranking. 

Discussion 

By applying ontologies to structure and maintain the 
knowledge base, we provided a flexible inclusion of 
different types of rules like necessary distance between 
objects or required access from a certain direction. 
Although the initial integration and domain-specific 
adjustments were complex and time-consuming, a 
reusable and easily extendable knowledge base was con- 
structed. Object-oriented aspects of ontologies particu- 
larly helped to formulate more general rules that applied 
to a variety of specialized concepts. One example of a 
general furniture item is a chair-concept in contrast to 
more specific items like a stool- or chair for blood with- 
drawal-concept All instances of a chair-concept must 
provide an accessible seat area which also applies to all 
inherited concepts. We look forward to reusing these 
ontologies in other projects that need flexible, extend- 
able and reusable knowledge bases. Using a game to 
teach interior design of a medical practice revealed some 
positive aspects. Common processes in a medical prac- 
tice are visualized and players can analyse and improve 



the interior design in the light of their findings. When 
integrating improvements, players are encouraged to ex- 
plore self-invented ideas. This is supported by the imme- 
diate feedback of simulated patients on new furniture 
and equipment arrangements. Also, players are naturally 
supported to train for their intended role as a manager 
of a medical practice: they observe and analyse the be- 
haviour of patients and deduce appropriate actions. 
According to reports by the educational supervisors of 
the gaming sessions the public player ranking in 



B 



2% 

Name: Dr. Cox 
I inspect a wound 




I 



Figure 10 Task information in simulation mode. 
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particular promoted valuable competitive collaborations 
and fostered motivated game play. Another motivating 
aspect of the game was the application of the circle of 
expertise. The three successive, sustainable and repeated 
steps of observation, analysis and improvement helped 
to create a motivating gaming experience throughout 
the game and helped to increase students' motivation to 
explore self-invented ideas. With regard to participants 
who react to stress and need individual support to mas- 
ter competitive situations we plan further investigations 



of this motivated gaming experience. As a further motiv- 
ating aspect we integrated amusing features into the 
game. For example, when players hover over an agent 
with the mouse in simulation mode, an information box 
appears that tells the player the name and current action 
of the agent (Figure 10). Each agent has a unique picture 
that shows characters from a famous TV show that is 
currently popular among medical students. According to 
reports by the educational supervisors these cheerful ele- 
ments fostered good mood and helped to create an 



Bitte bewerten Sie einzelne Eigenschaften der Lernsoftware 
zur Einrichtung einer Arztpraxis der Allgemeinmedizin 

Skala: 00 (sehr hoch) / ©@ (sehr niedrig) 



Eigenschaft 


Bewertung 


., 


(V 


gxg) 


1st die Software fur diese Zielgruppe geeignet? 


r 










1st die Ladezeit zufrieden stellend? 


r 


r 


r 


r 


r 


1st die Bildschirmaufteilung in Werkzeug- und Spielfeldbereich 

intuitiv 7i inanfilirh'? 


r 


r 


r 


r 


r 


1st die Gesamtgestaltung der Bildschirmoberflache 
ansprechend? 


r 


r 


r 


r 


r 


1st die Benutzerfuhrung selbsterklarend? 


□ 


□ 


n 


r 


n 


Konnten alle Texte gut gelesen werden? 


r 


c 


c 


r 


n 


Waren alle Symbole selbsterklarend? 


□ 


r 


r 


r 


n 


Konnte die Lernsoftware motivieren? 


r 


□ 


r 


r 


□ 


1st eine angenehme Lernatmosphare gewahrleistet? 


r 


r 


n 


r 


n 


Wird durch die Lernsoftware Neugierde gefordert? 


r 


□ 


r 


□ 


□ 


Bietet die Lernsoftware positive Uberraschungen? 


r 


□ 


r 


r 


□ 


Wird innerhalb der Lernsoftware ausreichend Humor geboten? 


r 


□ 


r 


r 


□ 


Fordert die Lernsoftware die Kreativitat? 


r 


r 


r 


r 


r 


1st die Schwierigkeit fur die Zielgruppe angemessen? 


r 


r 


r 


r 


r 


Gewahrleistet die Lernsoftware ausreichend Ruckmeldungen? 


r 


□ 


r 


r 


r 


Werden Lernziele ausreichend angegeben? 


r 


r 


r 


r 


r 


Sind die Lernziele fur alle Benutzer der Zielgruppe erreichbar? 


r 











Evaluation zur Lernsoftware emedoffice 

Planspiel: Praxisgrundung (Spieleinheit 4: Einrichtung einer 
Arztpraxis der Allgemeinmedizin) 



25. Februar 2010 



Figure 1 1 First page of usability questionnaire. 
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enjoyable learning atmosphere. Some items on the us- 
ability questionnaire of the first intervention were less 
well rated on average in comparison with the rest of the 
evaluated items. These are (a) Loading time, (b) Precise 
learning objective and (c) Symbol identification. Shortly 
after the evaluation the used machines were replaced by 
new ones which boosted the gaming speed and hence 
led to more sophisticated game play. Therefore a bad 



rating for (a) Loading time was compensated in the last 
evaluation. The rating for (b) Precise learning objective 
can potentially be increased by a short integrated intro- 
ductory game play video or a more detailed verbal descrip- 
tion of the learning objective and easy accomplishable 
sub-objectives. The third item (c) Symbol identification 
could be improved by an offer of more information if a 
tool in eMedOffice is used incorrectly. Additionally, the 







.V 

• 


- 




O© 


Bietet die Lernsoftware ausreichend Ubungsmoglichkeiten? 


r 


r 


r 


r 


r 


Stellt die Lernsoftware genugend Abwechslung bereit? 


r 


r 


r 


□ 


r 


Wird durch die Lernsoftware Wissen unterschiedlich, also 


r 


r 


r 


r 


r 


Faktenwissen oder Strategien zur Problemlosung, vermittelt? 






Wird Wissen vermittelt, wenn Benutzer nicht explizit 


r 


r 


r 


r 


r 


beabsichtigen zu lernen? 








Wird durch die Lernsoftware die Problemlosekompetenz 
gefordert? 


r 


c 


r 


□ 


r 



Bitte beschreiben Sie, was Ihnen an der Lernsoftware positiv aufgefallen ist. 



Bitte beschreiben Sie, was Ihnen an der Lernsoftware negativ aufgefallen ist. 



Haben Sie Ideen fur zukunftige Erweiterungen oder Verbesserungen? 



Vielen Dank fur Ihre Teilnahme an dieser Evaluation! 



Evaluation zur Lernsoftware emedoffice 25 - Februar 2010 

Planspiel: Praxisgrundung (Spieleinheit 4: Einrichtung einer 
Arztpraxis der Allgemeinmedizin) 

Figure 12 Second page of usability questionnaire. 
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introduction could be improved with respect to the in- 
game symbols. In future, there should be further evalua- 
tions that compare eMedOffice with ordinary courses that 
teach the same topic. Another open challenge is to prove 
the superiority of eMedOffice in contrast to other teach- 
ing methods, a necessary requirement for its enduring in- 
tegration in courses. 

Future work 

Given our experiences in the development and evalu- 
ation of eMedOffice, various extensions are possible. 
One important possible extension is the enhancement of 
the knowledge base. This includes the formalization of 
new rules to describe a good working medical practice. 
More rules would allow the expedient usage of different 
levels of difficulties and could enable more authentic 
scoring. Another important aspect is the integration of 
an internationalisation function that allows players to 
switch the language of eMedOffice. This is especially im- 
portant for international courses with many students 
from foreign countries. Moreover, additional languages 
could improve the subject-specific foreign languages of 
all students. An economic circuit could further extend 
the game to enhance players' commercial and financial 
understanding. Players could earn virtual money for each 
successfully treated patient. This earned money could be 
invested to further enhance the medical practice. In that 
context it could be an additional motivating aspect if all 
players competed for the same shared patient pool. 
Thus, more visits of patients would indicate a 'better' 



medical practice compared with a 'bad' one less fre- 
quently visited. This would leave more successful players 
with increasing difficulties and the need to satisfy more 
varied demands by the patients than less successful 
players who have more time to evaluate failed work- 
flows. Overall, it could benefit the balance of the game 
depending on each students performance. A shared pa- 
tient pool could also sensitize players to the growing 
competition for patients in the real world. Additionally, 
the evaluation elicited many valuable suggestions from 
players concerning improvements for the game inter- 
action like an enhanced method to rotate equipment on 
the ground plan or more detailed information about the 
placed furniture items. 

Conclusions 

The project eMedOffice implements a practical ap- 
proach for a serious game in a medical environment to 
teach optimization of design in terms of furnishing, 
equipment components and furniture items in a general 
medical practice. Designed as a serious game, it is inte- 
grated in the curricular courses of medical study at the 
RWTH Aachen University Medical School. By using 
motivating aspects like permanently visible public player 
ranking and fun elements, the game fostered valuable 
discussions and competitive collaboration among partici- 
pants. A statistical analysis of the results of an evaluation 
provides a clear indication of the positive learning effect 
of the serious game. A usability questionnaire survey 
reveals a very good overall score of 4.07 (5=best, 



Bitte bewerten Sie folgende Aussagen zur 
Einrichtung einer Arztpraxis der Allgemeinmedizin 

Skala: ©© (sehr hoch) / ©© (sehr niedrig) 



Kenntnis/Fahigkeit 


Einschatzung 


© 


© 


©© 


Ich kann wichtige Raume benennen. 




r 


r 




□ 


Ich kann wichtige Einrichtungsgegenstande benennen. 


r 


r 


r 


□ 


r 


Ich kann wichtige Ausrustungsgegenstande benennen. 


r 


r 


r 


r 


r 


Ich kann eine finanzielle Belastung fur eine Einrichtung 
einschatzen. 


r 


r 


r 


r 


r 


Ich weif3, welche Einrichtungsgegenstande welchen 
Raumen zugeordnet werden sollten. 


r 


r 


r 


r 


r 


Ich weiB, welche Ausrustung welcher Einrichtung in 
welchem Raum zugeordnet sein sollte. 


r 


r 


r 


r 


n 


Ich habe eine Idee von einem guten Arrangement von 
Einrichtungsgegenstanden. 













Evaluation zur Lernsoftware emedoff ice 25 - Febmar 201 0 

Planspiel: Praxisgrundung (Spieleinheit 4: Einrichtung einer Arztpraxis der Allgemeinmedizin) 



Figure 13 Self-report. 
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l=worst). In conclusion, we are confident that we can 
develop even more effective serious games in future that 
can be integrated into curricular courses of the RWTH 
Aachen University Medical School 

Availability and requirements 

Project name: eMedOffice 

Project home page: http://sourceforge.net/projects/emed 
office/ 

Operating system(s): Platform independent 
Programming language: PHP 
Other requirements: Apache2, mod-php, MySQL 
License: GNU GPL 

Any restrictions to use by non-academics: GNU GPL 
Usability questionnaire 

This section presents the usability questionnaire used to 
evaluate the quality of the proposed learning game eMe- 
dOffice (Figure 11 and Figure 12). As the evaluation 
took place at the RWTH Aachen Medical School in Ger- 
many, all contents are in German. 

Self-report 

This section provides the self-report sheet that partici- 
pants had to fill in before and after playing the learning 
game eMedOffice (Figure 13). 
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